Data Collection and Evaluation II

The first presentation, by Dave Pallett, described an evaluation by Pallett, Fiscus, and Garofolo (National Institute for Standards and Technology) of the significance of differences in performance of various speech understanding systems. A number of sites which are developing such systems have been using the DARPA Resource Management Speech Corpora to evaluate their systems. As performance gradually improves, and differences between systems and between successive versions of a system narrow, the question arises as to whether reported differences are statistically significant. To assess this question, the authors have implemented two statistical tests and applied them to the output of several speech understanding systems. They reported that, for speaker dependent systems and systems using a word-pair grammar, differences were often not significant; for speaker independent systems and systems not using a grammar, significant differences were found in many cases.